Medical diffusion on a budget

Textual Inversion for medical image generation

Bram de Wilde, Anindo Saha, Maarten de Rooij, Henkjan Huisman, Geert Litjens

🚧 TODO

  • add logo to title slide
  • bonus slide on chambon et al
  • bonus slide on FID metric

Diffusion models

Popular for text-to-image modeling

Stable Diffusion: open source + inference on single GPU

🚨 Training requires a lot of compute and data

🏥 Medical domain can have rare diseases and local compute

Prostate MRI?

🧐 Does Stable Diffusion already know medical imaging?

✒️ “a prostate MRI scan”

✒️ “a T2-weigthed MRI scan of a prostate”

💡 Fine-tune model

Fine-tuning diffusion models

Methods fine-tune sub-parts of diffusion model

✒️ Textual Inversion only trains token embedding

Textual Inversion

 

Adapt to medical imaging

Original work uses small embeddings and ~5 examples

💡 Use larger embeddings and more examples

Classification

🚧 image visualizing setup

Classification

#Real #Synthetic AUC ± std - Prostate MRI
200 0 0.780 ± 0.017
200 2000 0.803 ± 0.009

📈 Adding synthetic cases maintains or improves performance

Classification

#Real #Synthetic AUC ± std - Prostate MRI
200 0 0.780 ± 0.017
200 2000 0.803 ± 0.009
0 2000 0.766 ± 0.020

📈 Adding synthetic cases maintains or improves performance

📉 Using only synthetic data gives small performance drop

Classification

#Real #Synthetic AUC ± std - Prostate MRI
200 0 0.780 ± 0.017
200 2000 0.803 ± 0.009
0 2000 0.766 ± 0.020
0 20001 0.562 ± 0.036

📈 Adding synthetic cases maintains or improves performance

📉 Using only synthetic data gives small performance drop

👎 Training on 10-case embeddings shows quality difference

Comparison to GAN baseline

🧐 What about GANs?

🔬 Fine-tune a pre-trained StyleGAN3 on 100 images

⚖️ Similar training time and compute

🧑‍⚕️ Prostate radiologist preferred diffusion model (36/50)

Composing embeddings

↔︎️ Interpolate between healthy and diseased state

⚡ Train two embeddings: healthy, diseased

✒️ “healthy:30% AND diseased:70%”

Composing embeddings

➕ Combine multiple embeddings to show multiple diseases

⚡ Train embeddings per disease

✒️ “pleural_effusion AND pneumonia”

Disease inpainting

🖌️ Mask part of image and denoise

🔍 Precise control over disease location

Future directions

🩻 Fine-tune medical diffusion model

🧑‍💻 Applicable to 3D text-to-image models

🧫 Study utility for rare diseases

🛠️ Controlled synthesis with composing and inpainting

🧑‍💻 Try it out!

Scan for 📜 💾 ✉️ & more!

Chambon et al.

  • Bonus slide showing what happens if you have data + compute

FID metric

  • Highlight appendix